Method and apparatus for correcting crosstalk and spatial resolution for multichannel imaging

ABSTRACT

A multichannel imaging system generates an ensemble of images for each field of view of an object. Each image in the ensemble is intended to contain information from only one source among a plurality of sources for the object. However, due to crosstalk, at least a portion of the signal from a first source appears in a channel intended for a second source. Because the accuracy of the correction will be degraded if the images in an ensemble are spatially misaligned with respect to one another, the spatial offset between images is determined and a correction is applied to substantially eliminate the offset. Then, a correction to the signals is determined to substantially reduce the contributions to the signal in a channel from the signals in other channels. The signal processing can be employed to process the output signals for each of a plurality of different disclosed imaging systems.

RELATED APPLICATION

This application is a U.S. divisional patent application based on priorapplication Ser. No. 10/783,530, filed Feb. 24, 2004 now U.S. Pat. No.7,006,710, which itself is a U.S. divisional patent application based onprior application Ser. No. 10/132,059, filed Apr. 24, 2002 now U.S. Pat.No. 6,763,149, which in turn is a U.S. conventional patent applicationbased on prior provisional application Ser. No. 60/286,713, filed Apr.25, 2001, the benefits of the filing dates of which are hereby claimedunder 35 U.S.C. § 119(e) and 120.

FIELD OF THE INVENTION

The present invention generally relates to a method and apparatus forimproving the accuracy of quantitative images generated by multichannelimaging instruments, and more specifically, to correcting errorsintroduced by crosstalk between channels, with application to a broadrange of imaging instruments and particularly, to flow imaginginstruments using time-delay-integration image detectors.

BACKGROUND OF THE INVENTION

The parallel advancement of the technology of video microscopy andtechniques for preparing and staining biological samples has enabledthose working in areas such as fundamental biological science,diagnostic medicine, and drug discovery to gather an ever-increasingamount of information from a single biological specimen. In the fieldsof cell biology and clinical cytology, for example, specimens may bestained with absorption dyes to define cell morphology, and withfluorescent dyes that attach to molecules bound to specific proteins ornucleic acid chains. Microscopes equipped for exciting and imaging thefluorescent dyes and concurrently imaging cell structures are routinelyused for studying complex processes that modify cells on the grossstructural level and also at the molecular level. In recent years,computational analysis of images captured from multiparametermicroscopes has shown promise for automating large investigative studiessuch as those conducted by drug discovery and development companies andfor automating complex cellular diagnostic tests for clinical medicine.Optimal use of such technology can be attained only if the signals usedfor image generation are accurately scaled to information about thecells being studied. However, such information can be degraded duringthe capture process. Specifically, interference can be introduced into achannel dedicated to a first signal due to leakage of a signal intendedfor a second channel. This type of signal degradation is generallyreferred to as channel-to-channel crosstalk.

An advancement to computer-based multiparametric imaging is disclosed incommonly assigned U.S. Patents, both entitled IMAGING AND ANALYZINGPARAMETERS OF SMALL MOVING OBJECTS SUCH AS CELLS, U.S. Pat. No.6,249,341, issued Jun. 19, 2001 (filed Jan. 24, 2000), and U.S. Pat. No.6,211,955, issued Apr. 3, 2001 (filed Mar. 29, 2000), the completedisclosure, specification, and drawings of both of which are herebyspecifically incorporated herein by reference. The technology disclosedin these applications extends the methods of computer vision to theanalysis of objects either flowing in a fluid stream or moving relativeto the imaging instrument on a rigid substrate, such as a glass slide.Instruments based on the inventions of the patent applications citedabove deliver improved sensitivity at high spatial resolution throughthe use of time-delay-integration (TDI) electronic image acquisition, amethod wherein signal integration is accomplished by shifting chargepackets through an imaging array in synchrony with the motion of thetarget object being imaged.

The TDI-based flow imaging technology, with its ability to substantiallyimprove signal-to-noise ratio, is of exceptional utility formultiparametric imaging. Each of the channels of a TDI flow imaginginstrument can be dedicated to a single light source in the targetobjects. One such light source, for example, is the fluorescent dyeattached to a molecule selected for its specificity for binding to atarget protein. Each of a plurality of channels can be dedicated to aparticular different dye, and all of the dyes addressed by theinstrument may be present in close proximity on a single target cell.Because the dyes may have emission spectra broader than the passbands ofthe channels that collect their signals, channel-to-channel crosstalkcan prevent the accurate estimation of the intensity of the signal fromeach dye.

Accordingly, it would clearly be desirable to develop a method andapparatus that simultaneously offers speed and accuracy in eliminatingsuch channel-to-channel crosstalk. Preferably such crosstalk reductioncan be achieved in conjunction with the TDI-based flow imaging methodand apparatus noted above, which are intended for real time collectionand processing of images from objects moving in high concentration, athigh speed, through the instrument. Accordingly, the crosstalk reductionof the present invention is preferably applicable in real time and insynchrony with the collection of images of the moving targets thatinclude indicators attached to the targets.

SUMMARY OF THE INVENTION

The present invention is directed to enabling an accurate reconstructionof information about objects imaged by an instrument using multiplechannels, each channel being generally optimized to receive signals of atype differentiated from other signal types by predefinedcharacteristics. These predefined characteristics may include, but arenot limited to wavelength, a modulation of a signal received from asource, a scatter angle, a Doppler shift, and a phase shift (e.g., withrespect to a reference phase). The present invention applies to, but isnot limited to, instruments for collecting information fromelectromagnetic waves in all portions of the spectrum, by acousticwaves, by particle flux, and by measurement of object characteristicssuch as conductivity, chemical reactivity, size, shape, and mass.

One example of an application of the present invention is its use in amultiple-wavelength optical imaging instrument. In such an instrument,each channel is made sensitive to electromagnetic radiation ofwavelengths bounded by an upper and lower limit, defining differentwavebands for each channel. Typically these limits are determined by thecharacteristics of one or more filters disposed in a path between alight source and a photodetector servicing a channel. The images in eachchannel are detected, producing signals that are processed by thepresent invention to correct errors in alignment between the channelsand a reference and then, to correct for crosstalk between the channels.

Thus, the present invention is directed to a method and apparatus thatnot only corrects for crosstalk between channels, but also ensures thatsignal data in each channel is properly aligned with signal data inother channels, so that the benefit from the crosstalk correction is notdegraded by signal misalignment.

In one preferred embodiment, a method is provided for correcting signalmisalignment between individual channels in a multichannel imagingsystem, such that data in a first channel is substantially aligned withdata in other channels. The method also includes the step of reducingerroneous contributions to signal data from a source intended to providesignal data for other channels.

Preferably, the signal data are used to produce an image for display.Accordingly, a preferred embodiment is directed to a method thatincludes the step of spatially aligning images input in an imageensemble from a plurality of channels, such that each image in the imageensemble is substantially aligned with other images in the imageensemble, and the step of applying spectral crosstalk corrections, toremove the channel-to-channel crosstalk from the image ensemble output.

In one embodiment, the step of spatially aligning images includes thestep of utilizing two classes of information, including a first andsecond class of constants. The first class of constants includeshorizontal and vertical spatial offsets, which are derived from anon-line calibration image. The second class of constants is accessedduring the step of spatially aligning images, but is not modified.Preferably the second class of constants includes at least one ofchannel start columns for each image, and inverted source coefficients.

The horizontal and vertical spatial offsets are preferably generatedbased upon a comparison of each image in an image ensemble with acalibration image. The comparison with a calibration image can beperformed when a system for generating the multichannel signal isinitialized, and/or periodically during the use of a system forgenerating the multichannel signal.

The step of generating the horizontal and vertical spatial offsets caninclude the steps of detecting a boundary of an image, preparing acorrelogram based on the boundary and a reference image, determining apeak of the correlogram, and repositioning the image to correspond to apixel closest to the peak of the correlogram.

Preferably the horizontal and vertical spatial offsets are determinedfor each pixel of the image, and the detection of the boundary of animage involves the use of a two-dimensional gradient operator tosuppress flat surfaces and to enhance object boundaries. In oneembodiment, preparing the correlogram based on the boundary and thereference image involves preparing a correlogram in the spatial domain,while in other embodiments the correlogram is prepared in the frequencydomain.

In an embodiment in which the correlogram is prepared in the spatialdomain, a Fourier Transform is performed on boundary data for the imageand the reference image. Those results are multiplied to generate aproduct, and an inverse Fourier Transform is performed on that product.

To prepare the correlogram based on the boundary and the reference imagein the frequency domain, first a Fourier Transform is performed on theboundary data for the image and the reference image. Then a conjugationoperation is applied to one of the results of the Fourier Transforms.Next, the result of the conjugation operation is multiplied with theboundary data for the image to generate a product, and an inverseFourier Transform is performed on the product.

To minimize errors, groups of images in each data channel are preferablyprocessed together, such that a cumulative correlogram is generated foreach data channel.

Once the correlogram is complete, the peak of the correlogram definesthe aligned position of the image, relative to the reference imageemployed. The peak of the correlogram is determined by employing aTaylor series expansion, eigenvalues and eigenvectors. The image is thenmanipulated to align, to the nearest pixel, with that peak. Then, theimage is reconstructed by interpolating to a fraction of a pixel, toalign within a fraction of a pixel, with the true peak of thecorrelogram. Preferably, the interpolation involves the step of applyinga two-dimensional interpolation.

In one embodiment, the step of applying a two-dimensional interpolationincludes the step of computing a new amplitude value for each pixelbased on a weighted sum of a group of surrounding pixels. Preferably,the weighted sum is determined by a Taylor series expansion based on agroup of nine pixels, eight pixels of which surround a common originpixel. Coefficients are applied to each pixel value, and the sum of amatrix of the coefficients is equal to 1.0.

The step of reducing erroneous contributions to that channel'smeasurement by sources intended for other channels preferably involvessolving a set of linear equations relating source values to measurementvalues, wherein each channel is represented by a linear equation. It isalso preferred that the set of linear equations are solved for eachpixel in each image in each channel. The set of linear equationsrelating source values to measurement values can be solved by applyingsingular value decomposition to a matrix form of the set of linearequations.

The signal data, and/or corresponding images, can be spatially alignedin real-time. After the spatial alignment is completed, spectralcrosstalk corrections can also be applied in real-time, or after thesignal data/images have been stored for a period of time. The signaldata/images can also be spatially aligned after having been stored for aperiod of time.

Another aspect of the present invention relates to a method forcorrecting errors in a multichannel imaging system, wherein each channelis intended to contain signal information relating to an image of anobject that has been produced by only one type of source. The methodinvolves focusing light from the object along a collection path, anddispersing the light that is traveling along the collection path into aplurality of light beams, such that each light beam corresponds to asingle source. Each of the light beams is then focused to produce arespective image corresponding to that light beam. A detector isprovided, disposed to receive the respective images. The detectorgenerates an output signal corresponding to each image. For each outputsignal, a signal alignment correction and a crosstalk correction areapplied.

In addition to the aforementioned embodiments relating to the method,the present invention also relates to a system having elements thatcarry out functions generally consistent with the steps of the methoddescribed above. One such system relates to a multichannel imagingsystem for generating an ensemble of images from an object for eachfield of view, wherein each image in the ensemble contains informationfrom substantially only one type of source. The system includes acollection lens disposed so that light traveling from the object passesthrough the collection lens and travels along a collection path, and adispersing component disposed in the collection path so as to receivethe light that has passed through the collection lens, dispersing thelight into a plurality of separate light beams, each light beam beingdirected away from the dispersing component in a different predetermineddirection. The system also includes an imaging lens disposed to receivethe light beams from the dispersing component, thereby producing theensemble of images. The ensemble comprises a plurality of imagescorresponding to each of the light beams, each image being projected bythe imaging lens toward a different predetermined location Amultichannel detector is disposed to receive the plurality of imagesproduced by the imaging lens, and produces a plurality of outputsignals, such that a separate output signal is produced for each of theseparate light beams. Finally, the system includes means for processingeach output signal, wherein the means performs the functions ofcorrecting output signal misalignment between individual channels, suchthat an image generated by an output signal in each channel issubstantially aligned with a corresponding image in each other channel,reducing erroneous contributions to that channel's measurement bysources intended for other channels.

The system also preferably includes a display electrically coupled tothe means, the display generating an image for each output signal asmodified by the means. The means for processing preferably includes amemory in which a plurality of machine instructions defining a signalconditioning application are stored, and a processor that is coupled tothe memory to access the machine instructions, and to the display.Execution of the machine instructions by the processor cause it tospatially align images that are displayed, based on the output signalsfrom the multichannel detector, such that each image is substantiallyaligned with other images. The processor also applies the spectralcrosstalk corrections, to remove the channel-to-channel crosstalk fromthe displayed images.

It is further contemplated that the means for processing the signalalternatively comprise a programmed computer, an application specificintegrated circuit, or an oscilloscope.

Yet another embodiment of the system includes a plurality of differentdetectors, such that an image corresponding to a different source isdirected to each detector, and the plurality of different detectorscollectively comprise the multiple channels. The detectors employed inthis embodiment of the system are preferably pixilated. For example, aTDI detector can beneficially be employed to produce output signals byintegrating light from at least a portion of an object over time, whilerelative movement between the object and the imaging system occurs.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIG. 1A is a schematic diagram of an image collection and capture systemof a multichannel optical imaging instrument that includes a pluralityof cameras, with one camera and one filter per channel;

FIG. 1B is a schematic diagram of an image collection and capture systemfor an optical imaging system accomplishing multiparametric imaging witha single camera and a plurality of filters;

FIG. 2 is a graph of wavelength vs. intensity for three optical signalspectra and idealized passbands of corresponding bandpass filters;

FIG. 3 is a flow chart showing the logical steps generally implementedin the crosstalk correction method of the present invention;

FIG. 4 is a flow chart showing the logical steps in applying spatial andspectral corrections to image signals, in accord with the presentinvention;

FIG. 5 is a flow chart showing the logical steps in the calibrationphase of spatial corrections to image signals, in accord with thepresent invention;

FIG. 6 is a flow chart showing the steps employed for generatingaccumulated average correlograms between the data images and thereference image;

FIG. 7 is a schematic diagram indicating the locations of pixels used indetermining the correlogram gradient operator, and an equation for thatoperator;

FIG. 8 illustrates two surface plots and two contour plots of asubregion of an image, including one surface plot and one contour plotbefore, and one surface plot and one contour plot after application ofthe correlogram gradient operator;

FIG. 9 is flow chair showing the steps for computing the correlogrambetween a data image and a reference image in the frequency domain;

FIG. 10 is a schematic diagram depicting three data images and onereference image of objects in a flow stream, showing the objects asmisaligned;

FIG. 11 illustrates two grayscale images for two channels of a flowimaging instrument, indicating a vertical misalignment between the twochannels;

FIG. 12 is a surface plot of the cross-correlogram for the two imagesshown in FIG. 11;

FIG. 13 is a schematic diagram illustrating the relative locations andletter designations of pixels used in the computation of surfacederivatives for locating a peak of a correlogram;

FIG. 14 is a schematic diagram depicting the relative pixel locationsand variable names of the coefficients used in a two-dimensionalinterpolation for reconstructing the data images;

FIG. 15 illustrates surface plots of the interpolation coefficients fortwo pairs of image alignment offsets, including a zero offset correctionand an offset of 0.3 pixels (each axis);

FIG. 16 illustrates grayscale images showing three channels of amulti-parameter flow imaging system, both before and after theapplication of spatial and spectral corrections made in accord with thepresent invention;

FIG. 17 is a plan view of a first embodiment of the present invention inwhich particles conveyed by a fluid stream (at left side of Figure) aredepicted as flowing into the sheet;

FIG. 18 is a side elevational view of the first embodiment shown in FIG.17;

FIG. 19 is an isometric view of the first embodiment of FIG. 17;

FIG. 20 is an isometric view of a confocal embodiment that includes aslit for spatial filtering of extraneous light;

FIG. 21 is an isometric view showing different locations for a lightsource in connection with the first embodiment;

FIG. 22 is an alternative to the first embodiment in which a second setof imaging components and a second TDI detector is included formonitoring light from a particle, to avoid interference betweenFluorescent In Situ Hybridization (FISH) probes, and showing alternativelocations for light sources;

FIG. 23 is an isometric view of an embodiment in which an object issupported by a slide or substrate that moves past a collection lens,showing different locations for a light source;

FIGS. 24A and 24B are respectively a plan view and a side elevationalview of an alternative to the embodiment of FIG. 23, which is used toproduce a scattered pattern on the TDI detector;

FIG. 25 is a plan view of yet a further embodiment in which lightforming a scatter patterned image and spectrally dispersed light from anobject are imaged on separate portions of a TDI detector;

FIG. 26 is a plan view of a still further embodiment in which lightforming a scatter patterned image and spectrally dispersed light fromthe object are imaged on two different TDI detectors;

FIG. 27 is a plan view of an alternate embodiment that employs aspectral dispersion component comprising a plurality of stacked dichroicfilters employed to spectrally separate the light from an object;

FIG. 28 is a schematic illustration of a detection filter assembly thatmay optionally be disposed in front of the TDI detector in theembodiment of FIG. 27 to further suppress out-of-band light; and

FIG. 29 is a plan view of another embodiment of the configuration ofFIG. 27, wherein the spectral dispersion filter system comprises aplurality of dichroic cube filters orientated at various differentangles to create the spectral dispersing effect.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIGS. 1A and 1B illustrate two configurations of an instrument forimplementing the present invention. FIG. 1A shows an embodiment thatutilizes a plurality of photosensitive cameras 105. Lens 102 is used toform images on the photosensitive detector arrays of cameras 105. Thelight along each image formation path 101 is filtered by speciallydesigned mirrors 103 that transmit light in a first range of wavelengthsand reflect light in a second range of wavelengths, defining a pluralityof different wavebands that are received by individual cameras 105. Thesignals from cameras 105 are processed by signal processing means 106,which aligns the signals relative to each other, and reduces crosstalkbetween signals. An optional element is a display 107, on which theplurality of images corresponding to the processed signals can bedisplayed to a user.

While display 107 will often be beneficially incorporated in such asystem, such a display is not required. For example, a user may desireto collect data, process that data with signal processing means 106, andthen store the processed data for display at a later time. It is furthercontemplated that signal processing means 106 and display 107, which areenclosed in a block 109, can be employed to process data (i.e., amultichannel signal) that was previously generated and stored for aperiod of time. Of course, signal processing means 106 can be employedto process the multichannel signal at any time after the signal has beengenerated, and the processed signals (aligned and corrected forcrosstalk) can then be stored for a period of time before beingdisplayed on display 107 (or further processed). Thus, while the imagingsystems shown in FIGS. 1A and 1B represent exemplary preferredembodiments in which the present invention is applied, it should beunderstood that signal processing means 106 can be incorporated intomany other different types of systems that generate a multichannelsignal, or even used alone to process previously generated multichannelsignals.

Signal processing means 106 ensures that the signals from a multichannelsource are aligned relative to each other and reduces crosstalk amongthe signals from the multichannel source. The manner in which each ofthese functions is implemented is described in more detail below. Signalprocessing means 106 preferably comprises a programmed computing device,that includes a microprocessor and a memory in which machineinstructions are stored that cause the microprocessor to appropriatelyprocess the signals. Alternatively, the signal processing means cancomprise an application specific integrated circuit (ASIC) chip thatemploys hardwired logic for carrying out the processing functions, or adigital oscilloscope that can be configured to provide the requiredsignal processing capability.

An alternative configuration for an imaging system usable with thepresent invention is shown in FIG. 1B. In this configuration, a singlecamera 105 is used to form an image in which light from a plurality ofsources is filtered and reflected by a set of mirrors 103. Each mirrorreflects light in a different waveband, forming a plurality of images incorresponding different regions of the camera's photosensitive array.The light reflected by the first mirror in incident on a first region,while the light transmitted by the first mirror in the stack falls onthe face of the second mirror, which reflects light onto a second regionof the camera's photosensitive detector. The successive reflection andtransmission by the minor stack produces a plurality of spectrallyseparated images, and single camera 105 produces a multichannel signalcorresponding to those images formed on each region of the camera'sphotosensitive detector. These different signals are processed by signalprocessing means 106 and optionally displayed on display 107.

Preferably, the light being imaged by the imaging systems of FIGS. 1Aand 1B comprises wavelengths entirely encompassed within the passbandsof the various channels. In that case, each color of light, such as red,contributes to only one image (e.g., an image of an object that is thesource of the color of light in that channel). In many practicalapplications of the present invention, however, the light that formsimages in each channel spans a range of wavelengths broader than thepassband of an associated filter for the channel, as shown in FIG. 2. Inthis example, light 201 from each source is received in three channels202. The signal conveyed in each channel is then a composite ofinformation for the multiple sources. The object of the presentinvention is to process the signals from the plurality of channels todeliver the information for each source when displaying the image forthat source.

It will be understood that many other multichannel imaging systems canbenefit from the reduction in crosstalk provided by the presentinvention. FIGS. 17–29, which are described in detail below, illustrateexamples of additional embodiments of multichannel imaging systems withwhich signal processing means 106 can be employed, to reduce crosstalkbetween channels.

In optical imaging systems that produce images of an object, the objectmay modify incident light by absorption of specific wavelengths, byabsorption of the incident light followed by emission of light at adifferent wavelength, by diffraction of the incident light, or byrefraction of the incident light. An object can also emit light withoutbeing excited using incident light. Each channel of a multichannelimaging instrument is designed to produce images formed in response tothe light from an object produced in one of these ways. The presentinvention can be employed to enhance the contrast of such images.

In the present invention, the separation of signals from a detector intotheir proper channels is accomplished by solving a set of linearequations.s ₁=α₁₁ m ₁+α₁₂ m ₂+α₁₃ m ₃s ₂=α₂₁ m ₁+α₂₂ m ₂+α₂₃ m ₃, ands ₃=α₃₁ m ₁+α₃₂ m ₂+α₃₃ m ₃  (1)where m_(j) is a measurement from channel j, s_(i) is a characteristicparameter of source i, and α_(ij) is a weighting coefficient for sourcei into channel j.

These equations are solved using conventional methods of linear algebra,wherein the variables s_(i), and m_(j) are vectors and the variablesα_(ij) comprise a matrix. The general term for this process is“crosstalk correction,” since it removes the information from a sourcethat has spilled over into a channel adjacent to the correct channel.This information spillover is analogous to crosstalk in bundledtelephone lines, which results in a listener hearing a conversation fromanother line.

This set of equations must be solved for each pixel location in animage. It is essential that the images from all of the channels beprecisely aligned with one another so that the correct values areentered into the equations for each pixel location. Therefore, acomputational process is applied to the images corresponding to thefield of view before the crosstalk correction is applied. The “field ofview” is the scene captured in an image. The images belonging to a fieldof view are referred to herein as the “image ensemble” for that field ofview.

FIG. 3 illustrates the core processes carried out in the presentinvention and encompasses various signal processing stages, includingcalibrating for spatial correction parameters calibrating for spectralcorrection parameters and producing image ensemble outputs. The imageensemble outputs incorporate both a first spatial correction and asecond spectral correction. First, the spatial correction parameters aredetermined, then the spectral correction parameters are determined, andfinally image ensemble outputs are generated that incorporate both thespatial and spectral corrections. It should be noted that FIG. 3illustrates the data flow used in the present invention, as opposed to agraphical depiction of a sequence of logical steps.

The stage of calibrating for spatial correction parameters isrepresented by a block 301. In block 301, a calibration image isobtained from actual biological specimens, or artificial specimens (e.g.flow beads). In a preferred embodiment, the calibration imagecorresponds to a reference channel selected from among a plurality ofparallel channels. Most preferably, the reference channel is a brightfield channel, with the balance of the channels representingfluorescence channels. In such an embodiment, the calibration image fromthe reference channel is actually part of the ensemble of images. Thus,the spatial offsets are generated in real time. It is contemplated thata calibration image could be utilized before image ensemble datacorrelating to actual samples are generated. In such a case, the spatialoffset is not “live,” but instead is based on the offsets determinedwith respect to the different channels corresponding to a calibrationimage that is not part of the image ensemble in regard to sample data.Because such offsets are not determined “live,” it is anticipated thatoffsets determined in such an embodiment will not be as precise as thosedetermined when the calibration image is actually part of the imageensemble corresponding to a particular sample. This approach should beless computationally expensive, and can be beneficially employed whenprevious data corresponding to similar indicate that little error wouldbe introduced by using stored rather than live offset data.

Referring once again to FIG. 3, the algorithm computes the spatialcorrection parameters in a block 302. Once the spatial correctionparameters are computed, the spectral correction parameters aregenerated in a block 303. Note that the spectral calibration processrequires the use of a control, whereas any sample can be used to providespatial correction data. The control is preferably a biological specimenor an artificial specimen to which a single known fluorophore has beenattached. The fluorophore selected preferably has the characteristic ofhaving a fluorescent signature primarily limited to a single one of themulti-channels. Some small amount of “spill over” from the singlefluorophore will likely exist in the other channels. Based on the knownspectral signature of the control, and the multi-channel datacorresponding to that control, spectral corrections can be determined toeliminate such spill over, or crosstalk. Such a control is also referredto as a single source, because its spectral signature is substantiallyconfined to a single channel. A control can be imaged alone, as part ofa calibration phase that occurs before acquiring data from samples. Inat least one embodiment, a control is introduced into a batch ofsamples, so that the calibration can occur during the processing of abatch on samples.

Once the spatial and spectral correction factors have been determined,the signal processing can be performed on the ensemble of images. In ablock 304 the ensemble of images are input. The spatial corrections(determined in block 302) are applied to the ensemble of images in ablock 305. Next, the spectral crosstalk corrections determined in block303 are applied to the spatially corrected ensemble of images in a block306. It is important that the spatial corrections be applied before thespectral corrections are applied. The spatially and spectrally correctedensemble of images is available as data output at a block 307.

Note that during image processing (in blocks 304–307), the spatial andspectral computation algorithms represented by blocks 302 and 303,respectively, continue to produce measurements that indicate thestability or confidence of the corrections previously applied (withrespect to the “live” calibration embodiment described above). Once thealgorithm determines that such instability exists, the spatial andspectral calibrations can once again be performed, and the newlygenerated spatial and spectral correction factors can be applied toadditional ensemble of images (until instability is detected, which willcause additional calibrations to be performed). The image processingstages (of blocks 304–307) can be applied in real time as images arecollected during system operation, or offline, by accessing stored imagefiles. The images of the image ensemble output in block 307 can be usedfor further processing and analysis, free of the information degradationcaused by crosstalk between channels.

As noted above, stored data, rather than live data, can be used toprovide, or augment, the required spatial and spectral correctionfactors. In addition to stored data derived from a calibration image asdescribed above, other types of stored data can be used to provideadditional correction factors. Such data can include stored tables ofconstants that are derived from measurements and from knowncharacteristics of the objects being imaged. As noted in a block 308,the general positions or channel start columns for the images producedby an instrument such as that shown in FIG. 1B, and inverted sourcecoefficients, as indicated in a block 309, comprise stored informationthat can be used as correction factors. Such stored constants are notderived from a calibration image.

A flow chart of the general operations of an embodiment of the presentinvention is shown in FIG. 4. An image ensemble input in a block 401 isthe composite signal for a set of images, all of which depict the samefield of view, but each of which has been constructed using a signalfrom a different channel. The number of images in an ensemble is thusequal to the number of channels in the instrument. For example, an imageensemble in which there are four channels is represented in FIG. 10. Thereference channel can be any of the channels, although in one preferredembodiment, a bright field channel is preferred over fluorescencechannels for use as the reference channel. Referring to FIG. 10, forillustration purposes, the leftmost (first) channel is the referencechannel. The first image ensemble would thus include the first cellimage from each channel; the second image ensemble would include thesecond cell image from each channel; and the Nth image ensemble wouldinclude the Nth cell image from each channel.

The x (horizontal) and y (vertical) offsets 403 must be established inorder for the alignment process to operate on the image ensemble. Asnoted above in regard to block 301 of FIG. 3, the calibration image isprocessed to compute spatial X,Y offsets, determining offset values thatare input in a block 403 of FIG. 4. The calibration process may be runas a preliminary step in the operation of the instrument and ispreferably repeated periodically to accommodate any drift in the imageregistration caused, for example, by changes in temperature. Details ofthe process for generating the offsets are illustrated in the flow chartof FIG. 6.

In a block 404, the signals are aligned by selecting one signal in theimage ensemble as a reference image and shifting each other signal inthe image ensemble by pixel increments to align as closely as possibleto the reference signal. Note that shifting by whole pixel incrementswill rarely fully align the signals, so other steps are needed. For eachof the non reference signals, a 3×3 interpolation kernel is calculatedin a block 405. The interpolation process is described in more detailbelow, particularly with respect to FIG. 14.

Each non reference signal in the image ensemble is further aligned in ablock 406 to a fraction of a pixel by interpolation where the alignedimage is represented by expanded integer representation. Expanding theinteger representation of the image data to a larger number of bits thanthe resolution of the captured image enables faster processing to beachieved. This step is particularly important in embodiments thatgenerate spatial correction data “live.” In a preferred embodiment ofthe present invention, the image is originally captured with aresolution of 8-bits, and in block 406, the data are expanded to 16-bitsor 32-bits, for ease of processing. This step helps prevent theintroduction of round-off errors encountered when processing integershaving fewer bits.

Once each signal in the image ensemble is aligned, in a block 407 thecrosstalk correction is applied, using a spectral coefficient matrixfrom a block 408. The crosstalk correction is described in greaterdetail below. In a block 409, the expanded integer representation isreturned to the unexpanded integer representation corresponding to theresolution of the system. The purpose of the return to the unexpandedrepresentation is to reduce the memory required to store the processedimage data. In at least one embodiment, the unexpanded representation isan 8-bit digital gray-scale. In a block 402, the signals of the imageensemble have been aligned and crosstalk among the signals has beenreduced, and the processing of the image ensemble is complete.

Spatial Alignment

FIG. 5 summarizes the spatial calibration stage. A block 550 representsa sequence of image ensembles. A single image ensemble produces acorrelogram ensemble, where each correlogram corresponds to anon-reference channel convolved with the reference channel; thisconvolution is performed in a block 552. The correlogram ensemblegenerated from a single image ensemble may not accurately convey theimage offsets, especially if those images contain only a few objects.The accuracy and signal-to-noise ratio of the correlogram ensemble canbe enhanced by producing correlogram ensembles on a sequence of imageensembles. These correlogram ensembles are accumulated by the Nnon-reference channels in a block 554. Also in block 554, theaccumulated correlograms are used to generate a single correlogramensemble that represents the average of the sequence of correlogramensembles.

When the sequence of image ensembles have been processed, the exactpeaks of the accumulated correlogram ensemble are located by computingthe two-dimensional offsets in a block 556, and the coordinates of thepeaks are stored as the x and y offsets in a block 558. Since objectsmay occupy only a few pixels, and because the resolution of the imagingsystem may be on the order of a pixel width, alignment of one image tothe next, to a resolution of a fraction of a pixel width is necessaryfor accurate crosstalk correction. The true peak of the correlogram willrarely be centered on a pixel, but can be located by analyzing the shapeof the region around the pixel of maximum amplitude. By defining anaccurate equation for the correlogram amplitude in that region, the truelocation of the peak can be determined to a resolution that is afraction of a single pixel dimension. Therefore, the offset informationstored in block 158 consists of x and y offsets that are in units offractions of pixels. Then, an appropriate reconstruction filter isapplied to compute pixel values for the fractional offsets.

In summary, the spatial calibration process involves producing anaveraged correlogram ensemble and then computing the peak of the averagecorrelogram to within a fraction of a pixel position. Further detailsrelating to blocks 552 (Correlogram Ensemble) and 556 (OffsetDetermination) are provided below.

Correlogram Ensemble

FIG. 6 refers to the steps employed for computing the correlogramensemble. A block 602 represents the image ensemble input, and a block604 represents the reference image from the image ensemble. A block 606represents each non-reference data channel image. In a block 608, thereference image of block 604 is processed through a boundary enhancementalgorithm. The non-reference data channel images of block 606 are alsoprocessed through the boundary image enhancement algorithm in a block610. The boundary enhanced data from blocks 608 and 610 are used togenerate a correlogram in a block 612. The completed correlograms areaccumulated in a block 614, until a correlogram has been generated foreach data channel with the reference data channel. As described in moredetail below, several methods can be employed to generate thecorrelograms, including manipulating the signals in the frequency domainor in the spatial domain.

The boundary image enhancement performed in blocks 608 and 610 isachieved by using a two-dimensional gradient operator to suppress flatsurfaces and to enhance object boundaries to better facilitate boundarydetection. The operator builds signal in regions where the slope of theintensity is large in both the vertical and the horizontal directions.The linear expression for this operator is as follows:

$\begin{matrix}{G = {\frac{\partial}{\partial y}{\frac{\partial I}{\partial x}.}}} & (2)\end{matrix}$A preferred implementation of the gradient operator in sampled imagespace is as follows:G _(i,j)=|(I _(i+1,j+1) −I _(i−1,j+1))−(I _(i+1,j−1) −I_(i−1,j−1))|.  (3)

This operation is illustrated in FIG. 7. For each pixel in the image,the gradient is computed from the adjacent pixels, in the diagonaldirections. The effect of this gradient operator is illustrated in FIG.8. Surface plots 801 and 802 and contour maps 803 and 804 are shown foran image before and after transformation by the gradient operator. Theoperator transforms each Gaussian curve for an image into a cluster offour sharp peaks and is effective for images with either positive ornegative contrast.

Object boundaries carry all of the energy in the images transformed bythe gradient operator. It is then necessary to effectively overlay eachdata image onto the reference image and measure the vertical shift andhorizontal shift required to align object boundaries. Cross-correlationis a preferred method for measuring these shifts, or offsets, betweenthe reference and data images. As noted above, once the boundary datahave been generated, the correlogram is produced. The cross-correlationof two signals (s1 and s2), where one signal is a mirror of itself, isdetermined by convolving the two signals, producing a correlogram. Theconvolution operation in the spatial domain is defined by the followingequation:

$\begin{matrix}{{{f_{1}(t)} \otimes {f_{2}(t)}} = {\int_{- \infty}^{\infty}{{f_{1}(\lambda)}\;{f_{2}\left( {t - \lambda} \right)}\;{{\mathbb{d}\lambda}.}}}} & (4)\end{matrix}$Choosing the first function to represent the first signal, orf1(t)=s1(t), and the second function as the mirror of the second signal,or f2(t)=s2(−t), Equation (4) represents a correlogram in one dimension.The value of the convolution for every time sample, t, is the sum overinfinity of the product of the two functions, but with the secondfunction offset by a time delay t. The time delay between the twosignals can be determined by finding the peaks in the correlogram. Forimage realignment, the convolution is applied in two dimensions tofunctions in x, y space, as follows:

$\begin{matrix}{{{f_{1}\left( {x,y} \right)} \otimes {f_{2}\left( {x,y} \right)}} = {\int{\int_{- \infty}^{\infty}{{f_{1}\left( {ɛ,\eta} \right)}\mspace{11mu}{f_{2}\left( {{x - ɛ},{y - \eta}} \right)}\mspace{11mu}{\mathbb{d}ɛ}\mspace{11mu}{{\mathbb{d}\eta}.}}}}} & (5)\end{matrix}$For a two dimensional correlogram, let ƒ₁(x,y)=s₁(x,y) andƒ₂(x,y)=s₂(−x,−y). The alignment method is based on the premise thatthere will be similar structures in the images to be aligned. In theideal case, the second image is identical to the first except for ashift in x and y. Furthermore, the shift can be represented as theconvolution of the function with the two-dimensional Dirac deltafunction:ƒ₂(x,y)=ƒ₁(x−x ₀ ,y−y ₀), andƒ₂(x,y)=ƒ₁(x,y,)

δ(x−x ₀ ,y−y ₀).  (6)The image offsets, x₀ and y₀, can be measured from the shift in thetwo-dimensional correlogram, since:C _(1,2)=ƒ₁(x,y)

ƒ₂(x,y)=ƒ₁(x,y)

ƒ₁(x,y)

δ(x−x ₀ ,y−y ₀)  (7)

The Fourier Transform provides a convenient method for accomplishing theconvolution of two images, as evident from the relationship:ƒ₁(x,y)

ƒ₂(x,y)=F ⁻¹ [F ₁(ω_(x),ω_(y))·F ₂(ω_(x),ω_(y))],  (8)where F(ω_(x),ω_(y)) is the Fourier Transform of ƒ(x,y), andF⁻¹[X(ω_(x),ω_(y))] is the Inverse Fourier Transform of X(ω_(x)ω_(y)).The mirror of a function has the following relationship between spatialdomain and frequency domain representations:F ₂(ω_(x),ω_(y))={overscore (S ₂(ω_(x)ω_(y)))}=F┌ s ₂(−x,−y)┐where the overbar is the complex conjugation and S(ω_(x),ω_(y)) is theFourier Transform of s(x,y).

In the frequency domain, the two-dimensional convolution becomes anelement-by-element multiplication of the spectra of the two images.Therefore, a two-dimensional correlogram is obtained by performing aninverse Fourier Transform on an element-by-element multiplication of theFourier Transform on one signal and die Fourier Transform on the mirrorof the other signal. It should be noted that Equations 5–7 apply both tomanipulations in the frequency domain and the spatial domain, where asEquation 8 applies only to manipulations in the frequency domain.

In FIG. 9 are illustrated the steps for implementing the crosscorrelation process performed as a multiplication in the frequencydomain. Two images from the image ensemble are input to this process.The first image is a reference image, as indicated in a block 901, andis from the channel that is designated as the reference channel for thealignment. FIG. 10 depicts a four-channel system with a referencechannel 901. The other three channels 1002, 1003, and 1004 are datachannels. Preferably, channel 1001 corresponds to a bright field image,while channels 1002–1003 correspond to fluoresced images. As explainedbelow in further detail, the images from the data channels aretransformed to align vertically—as shown (i.e., in time) with thecorresponding images from the reference channel.

Referring again to FIG. 9, a complex Fourier Transform is applied toboth reference and data images in blocks 903 and 904, respectively. Theoutput of the Fourier Transform is a matrix of complex numbersrepresenting the amplitude and phase of each frequency in the base set.Before multiplication of the two spectra, a conjugation operation isapplied to the spectrum of the reference image in a block 905. Thecomplex conjugate of the spectrum of the reference image is multipliedby the spectrum of the data image in a block 906. Finally, an inverseFourier Transform is applied to the product in a block 908, whichproduces the two-dimensional correlogram.

While the process described above for preparing a two-dimensionalcorrelogram in the frequency domain is effective, manipulating thesignal data in the frequency domain is likely to be more computationallyexpensive than preparing a correlogram by manipulating the signal datain the spatial domain. The full range of offsets are computed in thefrequency domain, while knowledge about the sample set can be imputed tolimit the range of offsets computed in the spatial domain, as describedabove in conjunction with FIG. 6.

The computation of the correlogram in block 612 of FIG. 6 may beperformed either in the spatial domain or the frequency domain. If therelative spatial alignment cannot be restricted by prior knowledge, thenthe frequency domain is preferred for the sake of computationalefficiency. If, on the other hand, prior knowledge restricts thepossible alignments of the image channels to a relatively small range ofpositions, then it is more computationally efficient to compute thecorrelograms in the spatial domain.

In practice, it is likely that an instrument embodying the presentinvention would use the frequency domain method of computation during asystem set-up or qualification step, after which, the channel alignmentswould be fixed to a small range of possible positions, so that anyadjustments to the channel alignments for a given image ensemble wouldbe calculated in the spatial domain over this restricted range. FIG. 9,as discussed above, describes the calculation of the correlogram in thefrequency domain. The following discussion describes the calculation ofthe correlogram in the spatial domain.

As shown in FIG. 6, the calculation of the correlogram (regardless ofwhether the calculation is in the frequency domain or in the spatialdomain) uses two image inputs, one input (in block 604 of FIG. 6)corresponding to the reference channel, and the other input (in block606 of FIG. 6) corresponding to another data channel. Assign a label ofR_(ij) to the input data from the reference channel, and S_(ij) to theinput data from the other channel, where i and j are spatial indiceshaving integer values within the domain of each channel. Fordefiniteness, assume that in the current computation, i takes on valuesin the range [0, m−1] and j takes on values in the range [0, n−1]. Thecorrelogram value C_(a,b) is computed as a function of integer offsets aand b, by the following formula:

$\;{C_{a,b} = {\sum\limits_{i = 0}^{m - 1}{\sum\limits_{j = 0}^{n - 1}{R_{i,j}\; S_{{{({i + a})}\mspace{11mu}{mod}\mspace{11mu} m},{{({j + b})}\mspace{11mu}{mod}\mspace{11mu} n}}}}}}$where p mod q is the remainder when p is divided by q. When the maximumof the correlogram C_(a,b) is known to occur somewhere in a relativelysmall range of values of the offsets a and b, this formula may providethe most efficient computation of the correlogram available. If, on theother hand, the values C_(a,b) for the full range of possible offsets aand b must be computed, the computation in the frequency domain is muchmore efficient.

FIG. 11 shows two images 1101 and 1102 for two sets of objects. Thesecond set of objects in image 1102 is shifted along the Y axis(vertically—as shown) by an amount, Y_(error), relative to the first setof objects. FIG. 12 includes a surface plot 1201 of the correlogramgenerated using the process steps indicated in FIG. 9. Note that forperfect alignment of the two images, the peak of the correlogram wouldbe a single point at the origin of the plot.

Offset Determination

In the present invention, the correlogram peak region is described by asecond-order Taylor series expansion constructed from locally measuredfirst- and second-order derivatives. As an example, nine pixels centeredon a peak pixel, M, can be used to compute six derivatives. Referring toFIG. 13 for pixel locations 1301, the equations for these derivativesare as follows:dx=(A−B)/2dy=(C−D)/2d ² x=A+B−2Maxd ² y=C+D−2Maxdydx=dxdy=(E+F−G−H)/4  (9)

These terms consist of the first and second derivatives of the amplitudein the x and y directions and two terms that are more explicitlydescribed by partial derivatives as follows:

$\begin{matrix}\begin{matrix}{{{\mathbb{d}y}\;{\mathbb{d}x}} = {\frac{\partial}{y}\left( \frac{\partial A}{\partial x} \right)}} \\{and} \\{{{\mathbb{d}x}\;{\mathbb{d}y}} = {\frac{\partial}{\partial x}\left( \frac{\partial A}{\partial y} \right)}}\end{matrix} & (10)\end{matrix}$A matrix is constructed of the second-order derivatives as follows:

$\begin{matrix}{A = {\begin{bmatrix}{\mathbb{d}^{2}x} & {{\mathbb{d}y}\;{\mathbb{d}x}} \\{{\mathbb{d}x}\;{\mathbb{d}y}} & {\mathbb{d}^{2}y}\end{bmatrix}.}} & (11)\end{matrix}$The surface around the correlogram peak is then described by the Taylorseries expansion:C=C ₀ +{right arrow over (d)}•{right arrow over (p)}+0.5{right arrowover (p)}·A·{right arrow over (p)}  (12)where “•” is the Euclidian inner or dot product, {right arrow over (p)}is the vector from a peak pixel center to a true maximum, {right arrowover (d)} is a vector of first derivatives, A is a matrix of secondorder derivatives, and C₀ is the correlogram bias. The peak of thecorrelogram surface is identified as that location at which the gradientof the amplitude goes to zero, which corresponds to the followingcondition:Grad(C)={right arrow over (d)}+A·{right arrow over (p)}=0or:{right arrow over (p)}=A ⁻¹ ·{right arrow over (d)}  (13)

The two terms dxdy and dydx in matrix A are computationally identical.Therefore, A is symmetrical and orthogonally diagonalizable. The inverseof matrix A can therefore be conveniently found by computing theeigenvalues and eigenvectors of A and applying the followingequivalence:

$\begin{matrix}\begin{matrix}{A = {e^{t}\; D\; e}} \\{= {{\begin{bmatrix}e_{11} & e_{12} \\e_{21} & e_{22}\end{bmatrix}\begin{bmatrix}\lambda_{1} & 0 \\0 & \lambda_{2}\end{bmatrix}}\begin{bmatrix}e_{11} & e_{21} \\e_{12} & e_{22}\end{bmatrix}}}\end{matrix} & (14)\end{matrix}$where e in the first of the above equations is the eigenvector matrix ofA, and D equals the eigenvalue matrix of A. Similarly, the inverse of Ais:

$\begin{matrix}\begin{matrix}{A^{- 1} = {e^{t}\; D^{- 1}\; e}} \\{= {{\begin{bmatrix}e_{11} & e_{12} \\e_{21} & e_{22}\end{bmatrix}\begin{bmatrix}\frac{1}{\lambda_{1}} & 0 \\0 & \frac{1}{\lambda_{2}}\end{bmatrix}}\begin{bmatrix}e_{11} & e_{21} \\e_{12} & e_{22}\end{bmatrix}}}\end{matrix} & (15)\end{matrix}$

The inverse eigenvalue matrix D⁻¹ can be used in place of A⁻¹, if thederivative vector is rotated into the eigenvector coordinate system. Therotated version of the misalignment vector is as follows:

$\begin{matrix}{{\overset{\rightarrow}{p}}^{\prime} = {\frac{1}{\lambda}\;{\overset{\rightarrow}{d} \cdot {\overset{\rightarrow}{e}.}}}} & (16)\end{matrix}$Finally, the misalignment vector is rotated back into the image x, ycoordinate system by application of the transpose of the eigenvectormatrix, as follows:{right arrow over (p)}={right arrow over (p)}′· {right arrow over (p)}¹.  (17)The x offset o_(x) and y offset o_(y) to be applied to the data image torealign it with the reference image are then determined as follows:o _(x) =i _(max) +p _(x) and o _(y) =j _(max) +p _(y)  (18)where i_(max), j_(max) is the offset of the correlogram peak from theorigin, and p_(x), p_(y) corresponds to the projection of {right arrowover (p)} onto the x and y axes.

Typically, o_(x) and o_(y) are not integral multiples of pixels.Therefore, the data image cannot simply be shifted by a certain numberof pixels for alignment to the reference image. Instead, the data imageis aligned to the nearest pixel and then the content of the data imageis reconstructed by interpolation to a fraction of a pixel. The resultof this second operation closely approximates the amplitude values thatwould have been captured had there been no misalignment.

In the present invention, an image is reconstructed for alignment byapplication of a two-dimensional interpolation. In this process, oncethe image has been aligned with the reference image to the nearestpixel, the new amplitude value for each pixel is computed as theweighted sum of a group of surrounding pixels. The values of theweighting coefficients for the interpolation depend on the extent offractional pixel shifting that must be accomplished in the vertical andhorizontal directions.

For example, a new pixel value might be computed from a group of ninepixels as shown in FIG. 14. The center pixel is taken as the origin, andthe shift offsets, o_(x) and o_(y), define the desired new location forthe center pixel. A pixel that is far from the new location will have asmall value for its interpolation coefficient. A pixel that is veryclose to the desired location will have a large value for itsinterpolation coefficient.

In the preferred embodiment of the present invention, coefficients 1401and 1402 shown in FIG. 14 are related to the offsets, o_(x) and o_(y),through the following equations, which are derived from a Taylorexpansion model of the source intensity:

$\begin{matrix}\begin{matrix}{{{c\left( {0,0} \right)} = {1 - o_{x}^{2} - o_{y}^{2}}},} \\{{{c\left( {{- 1},1} \right)} = {{c\left( {1,{- 1}} \right)} = {- \frac{o_{x}o_{y}}{4}}}},} \\{{{c\left( {1,1} \right)} = {{c\left( {{- 1},{- 1}} \right)} = \frac{o_{x}o_{y}}{4}}},} \\{{{c\left( {0,1} \right)} = \frac{o_{y}\left( {o_{y} - 1} \right)}{2}},} \\{{{c\left( {0,{- 1}} \right)} = \frac{o_{y}\left( {o_{y} + 1} \right)}{2}},} \\{{{c\left( {{- 1},0} \right)} = \frac{o_{x}\left( {o_{x} - 1} \right)}{2}},{and}} \\{{c\left( {1,0} \right)} = \frac{o_{x}\left( {o_{x} + 1} \right)}{2}}\end{matrix} & (19)\end{matrix}$

FIG. 15 shows surface plots of the interpolation coefficient matricesproduced by the preceding system of equations for two pairs of offsetvalues, including an offset of zero (shown in a first surface plot1501), and an offset of (0.3, 0.3) (shown in a second surface plot1502). With the true correlogram peak exactly aligned to the centerpixel in the 3×3 region, as in first surface plot 1501, the coefficientmatrix is symmetrical about the center pixel. With the true peakresiding at (0.3, 0.3) in second surface plot 1502, the coefficients aregreater for the pixels nearest the true peak and less for the pixels farfrom the true peak.

The construction of the new, aligned data image is accomplished byapplying these coefficients to each pixel using the following equation:

$\begin{matrix}{{m^{\prime}\left( {i,j} \right)} = {\sum\limits_{p = {- 1}}^{1}{\sum\limits_{q = {- 1}}^{1}{{c\left( {p,q} \right)}{{m\left( {{i + p},{j + q}} \right)}.}}}}} & (20)\end{matrix}$

The coefficient matrix has the important property of summing to 1.0,independent of the location of the true peak relative to the pixel grid.If the system of equations for computing the interpolation coefficientsdid not have this property, the value of m′(i,j) would depend on theoffsets, o_(x) and o_(y), introducing inaccuracies in the imagereconstruction for some offset values.

Crosstalk Correction

In FIG. 3, once the data images have been thus aligned in block 305, thecrosstalk correction is carried out in block 306. The spatialcorrections must be applied before the crosstalk corrections are appliedto provide the maximum benefit. The emission spectrum plots of FIG. 2exemplify a typical problem to be solved when correcting crosstalk. In asystem involving fluorescent dyes and optical filters, the amplitudes ofthe measured signals for the three channels can be predicted from a setof equations, as follows:m ₁=α₁₁ s ₁+α₁₂ s ₂+α₁₃ s ₃m ₂=α₂₁ s ₁+α₂₂ s ₂+α₂₃ s ₃, andm ₃=α₃₁ s ₁+α₃₂ s ₂+α₃₃ s ₃  (21)where m_(i) is the measurement from channel i, s_(j) is the signal fromsource j, and α_(ij) is a weighting coefficient for source j intochannel i.

The same system of equations can be expressed using matrix notation, asfollows:{right arrow over (M)}=A{right arrow over (S)}.  (22)

where {right arrow over (M)} is a vector of measurements, A is acoefficient matrix, and {right arrow over (S)} is a vector of sources.

Preferably, the calibration process will be performed using objects(biological samples or synthetic beads) that only stimulate one of thespectral emission profiles (indicated by 201 within a single datachannel). As noted above, such samples or beads are referred to ascontrols. It is also possible to calibrate the instrument using a batchof different samples, if at least some of the samples are controls. Withrespect to such controls, each stimulus corresponds to one of the signalinputs s1, s2, or s3. Thus, when image ensembles are produced, theimages from each data channel will have a response (m1, m2, and m3) froma single stimulus (s1, s2, or s3). Using Equation (21), if one candetermine that the sample with spectral emissions produced in responseto the stimulus acting on the control (for example, stimulus s1) hasproduced the image ensemble, then coefficients from the first column ofmatrix A (Equation (22) are determined. Similarly, if one can determinethat the sample with spectral emissions corresponding to stimulus Nproduced an image ensemble, then the Nth column of matrix A isdetermined. Given the measured response, the determination of thestimulus source can be inferred from the spectral emissions in FIG. 2.The stimulus from channel N will produce the largest measurementresponse in image data channel N. In other words, the pixel values willalways be larger in the pixel from data channel N than the correspondingpixel location in other data channels. In the presence of noise, thesemeasurements of the columns of A are averaged over many sample inputs soas to improve the signal-to-noise ratio of the measurements. A typicalgoal when using a multichannel imaging system is to find the values ofvector S, since the S (source) values are usually related through somesimple formula to important intrinsic characteristics of the targetobjects. For example, if S is the intensity of light emitted byfluorescent molecules in a cell, and each different fluorescent moleculeis attached to a predetermined receptor protein molecule on the cellsurface, then repeated measurements of S for a large collection of cellscan be translated into an average number of receptor molecules per cell,which might well be a significant data point, e.g., in a biologicalexperiment. Because of crosstalk, the M (measurement) values will nothave a simple dependence on a single S value. However, techniques fromlinear algebra are available for solving the set of linear equationsrelating source values to measurement values.

The matrix form of the system of equations can be rearranged as follows:{right arrow over (S)}=A⁻¹ {right arrow over (M)}.  (23)In this equation, the source values are generated from the measurementsby matrix multiplication by the inverse of matrix A. The challenge is infinding that inverse.

The matrix A can reasonably be assumed to be square, since typically,one channel is dedicated to each source. However, this matrix cannot beassumed to be symmetrical. For that reason, the eigenvalue decompositioncannot be applied, and another method must be found for computing theinverse of A. Singular value decomposition is such a method. In thismethod, matrix A is expressed as the product of three matrices, asfollows:A=U^(t){right arrow over (D)}V  (24)where U, V equal row orthogonal matrices, U^(t) equals a transpose of U,and {right arrow over (D)} equals a vector of singular (diagonal) valuesof the matrix D.

Matrix D is the diagonal matrix (all off-diagonal components of D arezero) which results from the singular value decomposition of matrix A.The diagonal components of D are the “singular values” of the matrix A.Expressing an arbitrary matrix A in the form of Equation 24, where U andV are row orthonormal and D is diagonal, is referred to as “singularvalue decomposition.” As those of ordinary skill in the art willrecognize, using singular value decomposition readily facilitates thepopulation of these three matrices.

Singular value decomposition is applied to reduce crosstalk works asfollows. A vector M of measurements, and a vector S of the correspondingsource intensities are employed. Hypothetically, M and S are related byEquation 22, where A is the crosstalk matrix. The values of A may bedetermined either by modeling the system or by an empirical set ofmeasurements.

An important feature of this format for matrix A is that the inversionof matrix A is reduced to:A⁻¹=V^(t){right arrow over (D)}⁻¹U.  (25)The inversion operations are straightforward, consisting of transposingmatrix V and taking the inverse of the real number values of matrix D.With the new form of the inverse of the coefficient matrix, the equationfor the source values can be expanded into a form useful forcomputation, as follows:

$\begin{matrix}{\overset{->}{S} = {{A^{- 1}\overset{->}{M}}\mspace{14mu} = {V^{t}{\overset{->}{D}}^{- 1}U\overset{->}{M}}}} & (26)\end{matrix}$

Note that it is trivial to compute {right arrow over (D)}⁻¹ because D isa diagonal matrix, thus {right arrow over (D)}⁻¹ is merely the inverseof D. Note that M is the set of measurements generated by the imagingsystem, and that matrices V, D, and U are initially populated using thesingular value decomposition described above. Thus, the source values Scan be recovered, and crosstalk eliminated, from a set of measurements Mby applying Equation 26.

It should be noted that there are computationally less expensive ways tocompute the inverse of A. In a preferred embodiment, the aforementionedsingular value decomposition method is employed because singular valuedecomposition can be performed in a manner that ensures numericalstability. For example, the term {right arrow over (D)}⁻¹ in Equation 25is useful in determining this condition. The presence of some diagonalmatrix elements of D that are relatively small is an indication that theinversion of A is not numerically stable. Specifically, such a conditionindicates that there are one or more directions in source space, whichare nearly degenerate in measurement space. Under such circumstances, itis contemplated that it will be useful either to inform the user thatthe measurements being made include a degeneracy that cannot beaccurately resolved from the information provided, or that therelatively large values of {right arrow over (D)}⁻¹ (i.e., when valuesof D are relatively small, 1/D is relatively large) can be replaced withzero, to ignore the degenerate direction in the measurements.

Referring to Equation 26, the following three stages of computation areperformed. First, multiplication of the measurement vector by thetranspose of matrix V is implemented:

$\begin{matrix}{\begin{bmatrix}p_{1} \\p_{2} \\p_{3}\end{bmatrix} = {\begin{bmatrix}v_{11}^{t} & v_{12}^{t} & v_{13}^{t} \\v_{21}^{t} & v_{22}^{t} & v_{23}^{t} \\v_{31}^{t} & v_{32}^{t} & v_{33}^{t}\end{bmatrix}\begin{bmatrix}m_{1} \\m_{2} \\m_{3}\end{bmatrix}}} & (27)\end{matrix}$The “m_(x)” values are measurements from the signals, and the v′_(x)values are determined from matrix V, which is initially populated usingthe singular value decomposition described above.

Second, multiplication of the product vector by the inverse values ofthe singular values of matrix D is implemented:q₁=p₁d₁ ⁻¹q₂=p₂d₂ ⁻¹q₃=p₃d₃ ⁻¹  (28)where d_(i) ⁻¹ is the inverse of element i of {right arrow over (D)}.Note that the “p_(x)” values are obtained from equation (27), and thed_(x) ⁻¹ values are obtained from matrix D, which is initially populatedusing the singular value decomposition described above.

Third, multiplication by matrix U is implemented:

$\begin{matrix}{\begin{bmatrix}s_{1} \\s_{2} \\s_{3}\end{bmatrix} = {\begin{bmatrix}u_{11} & u_{12} & u_{13} \\u_{21} & u_{22} & u_{23} \\u_{31} & u_{32} & u_{33}\end{bmatrix}\begin{bmatrix}q_{1} \\q_{2} \\q_{3}\end{bmatrix}}} & (29)\end{matrix}$The “q_(x)” values are obtained from equation (28), and the u_(x) valuesare obtained from matrix U, which is initially populated using thesingular value decomposition described above.

FIG. 16 illustrates the effectiveness of the method of crosstalkcorrection of the present invention. Uncorrected images 1601 weresynthesized using the following system of equations:I ₁(i,j)=α₁₁ s ₁(i,j)+α₁₂ s ₂(i,j)+α₁₃ s ₃(i,j)I ₂(i,j)=α₂₁ s ₁(i,j)+α₂₂ s ₂(i,j)+α₂₃ s ₃(i,j), andI ₃(i,j)=α₃₁ s ₁(i,j)+α₃₂ s ₂(i,j)+α₃₃ s ₃(i,j)  (30)where I_(m)(i,j) is an image intensity from channel m at pixel i, j,s_(n), (i,j) is a signal from source n at pixel i, j, and α_(mn) is aweighting coefficient for source n into channel m.

Each object in the synthesized images is assigned to a particular sourceand appears in only one channel, in the absence of crosstalk. Acomparison of corrected images 1602 with uncorrected images 1601 showsthat the correction separates the objects into their respective channelswith very little residual error.

Additional Exemplary Preferred Embodiments of Imaging Systems

FIGS. 17–30 and the following descriptions disclose various embodimentsof imaging systems and detectors that can be employed to generatemultichannel image data, which then can be processed using the methoddescribed above to reduce crosstalk among the plurality of channels. Itshould be noted that none of these embodiments of imaging systemsspecifically show signal processing means 106 (see FIGS. 1A and 1B).However, it will be understood that signal processing means 106 isreadily coupled to the detectors of FIGS. 17–30 to enable the signalsgenerated by such detectors to be processed in accord with the presentinvention.

A first additional preferred embodiment of an imaging system 20 inaccord with the present invention is schematically illustrated in FIGS.17, 18, and 19, in connection with producing images of moving objectssuch as cells that are conveyed by a fluid flow 22 through the imagingsystem. In FIG. 17, fluid flow 22 entrains an object 24 (such as a cell,but alternatively, a small particle) and carries the object through theimaging system. The direction of the fluid flow in FIG. 17 is into (orout of) the sheet, while in FIGS. 18 and 19, the direction of flow isfrom top to bottom, as indicated by the arrow to the left of theFigures. Light 30 from object 24 passes through collection lenses 32 aand 32 b that collect the light, producing collected light 34, which isapproximately focussed at infinity, i.e. the rays of collected lightfrom collection lens 32 b are generally parallel. Collected light 34enters a prism 36, which disperses the light, producing dispersed light38. The dispersed light then enters imaging lenses 40 a and 40 b, whichfocuses light 42 onto a TDI detector 44.

As will be evident in FIG. 18, if the Figure depicts the imaging ofobject 24 over time, the object is shown at both a position 26 and aposition 28 as it moves with fluid flow 22. As a consequence, images ofobject 24 will be produced on the detector at two discrete spatialpositions 26′ and 28′, as indicated on the right side of FIG. 18.Alternatively, if FIG. 18 is depicting a single instant in time,positions 26 and 28 can represent the location of two separate objects,which are simultaneously imaged on the detector at positions 26′ and28′.

In regard to imaging system 20 and all other imaging systems illustratedherein, it will be understood that the lenses and other optical elementsillustrated are shown only in a relatively simple form. Thus, thecollection lens is illustrated as a compound lens comprising onlycollection lenses 32 a and 32 b. Lens elements of different designs,either simpler or more complex, could be used in constructing theimaging system to provide the desired optical performance, as will beunderstood by those of ordinary skill in the art. The actual lenses oroptical elements used in the imaging system will depend upon theparticular type of imaging application for which the imaging system willbe employed.

In each of the embodiments of the present invention, it will beunderstood that relative movement exists between the object being imagedand the imaging system. In most cases, it will be more convenient tomove the object than to move the imaging system. However, it is alsocontemplated that in some cases, the object may remain stationary andthe imaging system move relative to it. As a further alternative, boththe imaging system and the object may be in motion but either indifferent directions or at different rates.

The TDI detector that is used in the various embodiments of the presentinvention preferably comprises a rectangular charge-coupled device (CCD)that employs a specialized pixel readout algorithm, as explained below.Non-TDI CCD arrays are commonly used for two-dimensional imaging incameras. In a standard CCD array, photons that are incident on a pixelproduce charges that are trapped in the pixel. The photon charges fromeach pixel are read out of the detector array by shifting the chargesfrom one pixel to the next, and then onto an output capacitor, producinga voltage proportional to the charge. Between pixel readings, thecapacitor is discharged and the process is repeated for every pixel onthe chip. During the readout, the array must be shielded from any lightexposure to prevent charge generation in the pixels that have not yetbeen read.

In one type of TDI detector 44, which comprises a CCD array, the CCDarray remains exposed to the light as the pixels are read out. Thereadout occurs one row at a time from the top toward the bottom of thearray. Once a first row is read out, the remaining rows are shifted byone pixel in the direction of the row that has just been read. If theobject being imaged onto the array moves in synchrony with the motion ofthe pixels, light from the object is integrated for the duration of theTDI detector's total readout period without image blurring. The signalstrength produced by a TDI detector will increase linearly with theintegration period, which is proportional to the number of TDI rows, butthe noise will increase only as the square root of the integrationperiod, resulting in an overall increase in the signal-to-noise ratio bythe square root of the number of rows. One TDI detector suitable for usein the present invention is a Dalsa Corp., Type IL-E2 image sensor,although other equivalent or better image sensors can alternatively beused. The Dalsa image sensor has 96 stages or rows, each comprising 512pixels; other types of image sensors useable in the present inventionmay have different configurations of rows and columns or anon-rectilinear arrangement of pixels. The Dalsa sensor hasapproximately 96 times the sensitivity and nearly 10 times thesignal-to-noise ratio of a standard CCD array. The extended integrationtime associated with TDI detection also serves to average out temporaland spatial illumination variations, increasing measurement consistency.

In imaging system 20 and in other embodiments of the present inventionthat employ a fluid flow to carry objects through the imaging system, aflow-through cuvette or a jet (not shown) contains the cells or otherobjects being analyzed. The velocity and cellular concentration of thefluid may be controlled using syringe pumps, gas pressure, or otherpumping methods (not shown) to drive a sample solution through thesystem to match the pixel readout rate of the TDI detector. However, itshould be understood that the readout rate of the TDI detector could beselectively controlled, as required, to match the motion of the samplesolution.

Various optical magnifications can be used to achieve a desiredresolution of the object that is being imaged on the light sensitiveregions (pixels) of the TDI detector. It is contemplated that in mostembodiments, the optical magnification will fall within a range of 1:1to 50:1, providing a substantial range in the number of light sensitiveregions on the TDI detector on which images of the object are formed,also depending of course, on the actual size of the object being imagedand its distance from the imaging system. It is envisioned that thepresent invention can have applications ranging from the analysis ofcells and other microscopic objects to the imaging of stellar objects.

It should be emphasized that the present invention is not limited to CCDtypes of TDI detectors. Other types of TDI detectors, such ascomplementary metal oxide semiconductor (CMOS) and multichannel plateimaging devices might also be used for the TDI detector in the presentinvention. It is important to understand that any pixilated device(i.e., having a multitude of light sensitive regions) in which a signalproduced in response to radiation directed at the device can be causedto move through the device in a controlled fashion is suitable for useas the TDI detector in the present invention. Typically, the signal willmove in synchrony with a moving image projected onto the device, therebyincreasing the integration time for the image, without causing blurring.However, the motion of the signal can be selectively desynchronized fromthe motion of the radiation image, as required to achieve a desiredaffect.

FIG. 20 illustrates an imaging system 45, which is similar in many waysto imaging system 20. However, imaging system 45 is a confocalembodiment that includes a plate 50 having a slit 52 that substantiallyprevents extraneous light from reaching TDI detector 44. In imagingsystem 45, light 46 from object 24 is focussed by an objective lens 48onto a slit 52. Slit 52, as shown in FIG. 20, is sufficiently narrow toblock light that is not focussed onto the slit by objective lens 48 frompassing through the slit. Light 30′ passes through the slit and iscollected by collection lens 32 as discussed above, in regard to imagingsystem 20. Collected light 34 is spectrally dispersed by prism 36, andis imaged by imaging lens 40 onto TDI detector 44, also as discussedabove. By excluding light other than that from object 24 from reachingTDI detector 44, the TDI detector produces an output signal thatcorresponds only to the actual images of the object, and the signal isnot affected by the extraneous light, which has been excluded. If notexcluded in this manner, the ambient light reaching TDI detector 44might otherwise produce “noise” in the output signal from the TDIdetector.

It should be noted that in the illustration of each of imaging systems20 and 45, a light source has not been shown. These first twoembodiments have been illustrated in their most general form to makeclear that a separate light source is not required to produce an imageof the object, if the object is luminescent, i.e., if the objectproduces light. However, many of the applications of the presentinvention will require that one or more light sources be used to providelight that is incident on the object being imaged. The location of thelight sources substantially affects the interaction of the incidentlight with the object and the kind of information that can be obtainedfrom the images on the TDI detector.

In FIG. 21, several different locations of light sources usable toprovide light incident on object 24 are illustrated. It should beunderstood, however, that light sources could be located at many otherpositions besides those shown in FIG. 21. The location of each one ormore light source employed will be dependent upon the kind of imaging ofthe object, and the kind of data for the object, to be derived from thesignal produced by the TDI detector. For example, employing a lightsource 60 a or a light source 60 b, as shown in the Figure, will providelight 58 that is incident on object 24 and which is scattered from theobject into the optical axis of collection lens 32. The optical axis ofcollection lens 32 is at about a 90° angle relative to the directions ofthe light incident upon object 24 from either light source 60 a or 60 b.

In contrast, a light source 62 is disposed so that light 58 emitted fromthe source travels toward the object in a direction that is generallyaligned with the optical axis of collection lens 32, so that the imageformed on TDI detector 44 will not include light absorbed by object 24.Light absorption characteristics of the object can thus be determined byilluminating the object using a light source 62.

A light source 64 is disposed to illuminate object 24 with lightdirected toward the object along a path that is approximately 30–45° offthe optical axis of collection lens 32. This light 58, when incident onobject 24 will be reflected (scattered) from object 24, and thereflected or scattered light will be imaged on TDI detector 44. A moredirectly reflected light is provided by an epi light source 66, disposedso as to direct its light 58 toward a partially reflective surface 68that is disposed so that a portion of the light is reflected throughcollection lens 32 and onto object 24. The light reaching the objectwill be reflected from it back along the axis of collection lens 32 andwill at least in part pass through partially reflective surface 68 toform an image of the object on TDI detector 44. Alternatively, adichroic mirror may be employed instead of, and in the position of,partially reflective surface 68 to direct light from epi light source 66to excite fluorescence or other stimulated emission from object 24.Emission from object 24 is then at least partially collected bycollection lens 32 and passes through the dichroic mirror for spectraldispersion and detection by the TDI detector.

In addition to imaging an object with the light that is incident on it,a light source can also be used to stimulate emission of light from theobject. For example, FISH probes that have been inserted into cells willfluoresce when excited by light, producing a correspondingcharacteristic emission spectra from any excited FISH probe that can beimaged on TDI detector 44. In FIG. 21, light sources 60 a, 60 b, 64, or66 could alternatively be used for causing the excitation of FISH probeson object 24, enabling TDI detector 44 to image FISH spots produced bythe FISH probes on the TDI detector at different locations as a resultof the spectral dispersion of the light from the object that is providedby prism 36. The disposition of these FISH spots on the TDI detectorsurface will depend upon their emission spectra and their location inthe object. Use of FISH probes in connection with producing images ofFISH spots on the TDI detector with the present invention is discussedin greater detail below.

Each of the light sources illustrated in FIG. 21 produces light 58,which can either be coherent, non-coherent, broadband or narrowbandlight, depending upon the application of the imaging system desired.Thus, a tungsten filament light source can be used for applications inwhich a narrowband light source is not required. For applications suchas stimulating the emission of fluorescence from FISH probes, narrowbandlaser light is preferred, since it also enables a spectrally-decomposed,non-distorted image of the object to be produced from light scattered bythe object. This scattered light image will be separately resolved fromthe FISH spots produced on TDI detector 44, so long as the emissionspectra of any FISH spots are at different wavelengths than thewavelength of the laser light. The light source can be either of thecontinuous wave (CW) or pulsed type. If a pulsed type illuminationsource is employed, the extended integration period associated with TDIdetection can allow the integration of signal from multiple pulses.Furthermore, it is not necessary for the light to be pulsed insynchronization with the TDI detector.

Pulsed lasers offer several advantages over CW lasers as a light sourcein the present invention, including smaller size, higher efficiency,higher reliability, and the ability to deliver numerous wavelengthssimultaneously. Another advantage of pulsed lasers is their ability toachieve saturating levels of fluorescence excitation of fluorescentprobes used in cells. Fluorescence saturation occurs when the number ofphotons encountering a fluorescent molecule exceeds its absorptioncapacity. Saturating excitation produced by a pulsed laser is inherentlyless noisy than unsaturating CW laser excitation because variations inpulse-to-pulse excitation intensity have little effect on thefluorescence emission intensity.

Prism 36 in the imaging systems discussed above can be replaced with adiffraction grating, since either is capable of spectrally dispersingthe optical signals from the cells over the pixels of the TDI detector.In addition to providing useful data from a cell or other object,spectral dispersion can be used to reduce measurement noise. In caseswhere the light source wavelength differs from the emission spectra ofthe fluorescent probes, the light from the source that is scattered intothe collection system is spatially isolated from the fluorescencesignals. If the light source wavelength overlaps the emission spectra ofthe fluorescent probes, the pixels of the TDI detector in which light ofthe wavelength of the source falls can be isolated from those pixels onwhich the remaining fluorescence signals fall. Further, by dispersingthe fluorescence signals over multiple pixels, the overall dynamic rangeof the imaging system is increased.

Third Additional Exemplary Preferred Embodiment

A third additional preferred embodiment of an imaging system with whichthe present invention is useful is directed to a stereoscopicarrangement, as illustrated in FIG. 22. This arrangement enables theimaging of the object from two different directions in order todistinguish features that would otherwise overlap when viewed from asingle direction. While the third preferred embodiment can be employedfor objects on moving substrates such as microscope slides, it isparticularly useful for analyzing multicomponent objects in solution,such as cells containing FISH probes. Such probes appear as pointsources of light anywhere within the cell's three-dimensional nucleus.In some cases, two or more FISH probes may appear in an overlappingrelationship along the optical axis of the imaging system. In suchcases, one of the FISHY probes may obscure the others, making itdifficult to determine the number of probes present in the cell. Thisproblem of overlapping is a key factor in the determination of geneticabnormalities such as trisomy, otherwise known as Down's syndrome.Single-perspective systems may address this problem by “panning through”the object along the optical axis to acquire multiple image planes inthe object. While this approach may be effective, it requires asignificant amount of time to collect multiple images and cannot bereadily applied to a cell in flow. The stereoscopic imaging system 70 inFIG. 22 includes two TDI detectors 44 a and 44 b, and their associatedoptical components, as discussed above in connection with imaging system20.

By positioning the optical axes of collection lenses 32 for the two TDIdetectors so that they are spaced apart, for example, by 90°, it ispossible to separately resolve the FISH spots imaged from two or moreFISH probes on at least one of TDI detectors 44 a or 44 b. If two ormore FISH probes overlap in regard to the image produced on one of thedetectors, they will be separately resolved in the spectrally dispersedimages produced on the other TDI detector. Further, the use of two TDIdetectors in imaging system 70 in what might be referred to as a “stereoor three-dimensional configuration” allows flexibility in theconfiguration of each leg of the system, including parameters such asthe relative TDI readout rates, axial orientations, inclinations, focalplane positions and magnification. Multiple cells or other objects maybe imaged onto each detector simultaneously in the vertical direction.Since the objects may move in synchronicity with the signal on the TDI,no gate or shutter is required to prevent blurring of the image. Aspreviously noted, the present invention can use a pulsed or CW lightsource without need for a trigger mechanism to time a pulse coincidentwith particle arrival in the field of view. If a pulsed light source isused, the extended field of view in the axis of motion associated withTDI detection allows the cell or object in motion to be illuminated bymultiple pulses during its traversal. In contrast to a frame-basedimaging apparatus, a TDI system can produce a single unblurred image ofthe object that integrates the signal from multiple pulses. When a CWlight source is used, the signal generated by the object will becollected throughout the entire traversal of the object through thefield of view, as opposed to only a small segment in time when a shutteris open. Therefore, the amount of signal collected and imaged on thedetector in the present invention is substantially greater than that ofthe prior art frame-based imaging systems. Consequently, the presentinvention can operate at very high throughput rates with excellentsignal to noise ratio.

Also illustrated in FIG. 22 are several exemplary positions for lightsources, which are useful for different purposes in connection with theimaging system illustrated therein. In connection with TDI detector 44a, light source 62 provides illumination of object 24 from a directionso that absorption characteristics of the object can be determined fromthe image produced on the TDI detector. At the same time, light providedby light source 62 that is scattered from object 24 can be used toproduce a scatter image and spectrally dispersed images on TDI detector44 b. Light source 74 can be employed to produce spectrally dispersedand scattered images on both TDI detectors 44 a and 44 b. If lightsources 62 and 72 are of different wavelengths and an appropriate filteris provided to block the wavelength from the light source aligned withthe optical axis of the respective collections lenses 32, these twolight sources can be used for producing scattered light from the object.For example, suppose light source 72 produces light of a wavelength Athat scatters from object 24 and is directed toward TDI detector 44 a.By including a filter (not shown) that blocks wavelength B produced bylight source 62, the light at wavelength B will not directly affect theimages produced on TDI detector 44 a. Similarly, the light from lightsource 72 would be blocked with an appropriate filter (not shown) sothat it does not interfere with the imaging of light produced by lightsource 62 that is scattered from object 24 onto TDI detector 44 b.

Epi light source 66 is also illustrated for use in producing images onTDI detector 44 a in conjunction with partial reflector 68. Light source64 can be used to generate reflected light to produce images on TDIdetector 44 a, while scattered light from this source is directed towardTDI detector 44 b. These and other possible locations of light sourceswill be apparent to those of ordinary skill in the art, as appropriatefor providing the incident light on the object needed to achieveimaging, depending upon the particular application and information aboutthe object that is desired.

Imaging Slide or Object Carried by Slide

Turning now to FIG. 23, an imaging system 80 is illustrated that issimilar to imaging system 20, except that it is used for imaging object24 on a slide 82 (or other moving substrate). Object 24 is supported byslide 82 and the slide moves relative to the imaging system as shown inFIG. 23. Alternatively, slide 82 may be the object that is imaged. Forexample, the object may be a semiconductor wafer, paper, or other objectof interest, since the object may be imaged using reflected incidentlight.

To provide light incident on either slide 82 or object 24 that issupported thereby, a light source can be placed at one of severaldifferent locations. Exemplary light sources 62, 64, and 66 illustratesome of the locations at which light sources useful in this embodimentmay be disposed. Light 58 emitted by any of the light sources can beeither coherent or non-coherent light, pulsed or CW, and can be directedthrough slide 82 (if it is transparent) from light source 62 or can bereflected from the object or slide, if light sources 64 or 66 areemployed. As noted previously, epi light source 66 illuminates theobject in connection with a partially reflective surface 68.

FIGS. 24A and 24B show two different views of yet another preferredembodiment, which is an imaging system 90 that produces a scatteredpattern image of object 24 on TDI detector 44. Light 30 from object 24passes through collection lenses 32 a and 32 b, and collected light 34is directed onto a cylindrical lens 92, as in the previous embodiments.Cylindrical lens 92 focuses light 94 on TDI detector 44, generally alonga line that is aligned with a central axis 96 of cylindrical lens 92.Central axis 96 is shown in FIG. 24B, and it will be apparent that it isorthogonal to the direction in which object 24 moves through the imagingsystem. As object 24 moves downwardly, relative to its disposition asshown in FIG. 24A, the focus of cylindrical lens 92 on TDI detector 44moves upwardly. Cylindrical lens 92 thus distributes an image of theobject along a row or rows of the light sensitive regions or pixels ofTDI detector 44.

Referring now to FIG. 25, an illustration is provided of an imagingsystem 170 that produces both a scattered pattern image and a spectrallydispersed image of object 24 on TDI detector 44. In imaging system 170,light 30 from object 24 passes through collections lenses 32 a and 32 b,which produce infinitely focussed light 34 directed toward a dichroicfilter 172. Dichroic filter 172 reflects light of a specific wavelength,e.g., the wavelength of a light source (not shown) that is incident uponobject 24. Light of any other wavelength is transmitted through dichroicfilter 172 toward a diffraction grating 182. Diffraction grating 182spectrally disperses the light transmitted through dichroic filter 172,which typically would be light produced by the fluorescence of FISHprobes on object 24, so that a plurality of FISH spots corresponding tothe number of different FISH probes and objects being imaged areproduced on TDI detector 44.

Light 174, which is reflected from dichroic filter 172, is transmittedinto cylindrical lens 176 yielding focused light 178 directed along aline as a scattered pattern image in a region 180 on the TDI detector.The spectrally dispersed images of FISH spots or other aspects of object24 having wavelengths different than that reflected by dichroic filter172 are imaged as light 186 by imaging lenses 184 a and 184 b onto aregion 188 of the TDI detector. Thus, signals corresponding to thescattered pattern image and the spectrally dispersed images are bothproduced by TDI detector 44.

In FIG. 26, an imaging system 190 is illustrated that is slightlydifferent than the preceding embodiment, since a dichroic filter 172′ isemployed that is angled in a different direction, toward a second TDIdetector 44 b. A dispersed pattern image represented by light 178′ isproduced by a cylindrical lens 176′ in this embodiment. Just as inimaging system 170, light transmitted through dichroic filter 172′ isfocused onto TDI detector 44 a. Aside from using two separate TDIdetectors that are disposed at different sides of the imaging system,imaging system 190 is substantially identical in operation to imagingsystem 170. However, just as in the third preferred embodiment, the useof two separate TDI detectors allows flexibility in the configuration ofeach leg of the system, including parameters such as the relative TDIreadout rates, axial orientations, inclinations, focal plane positions,and magnification. It should also be noted that imaging system 170 couldbe constructed to include two separate TDI detectors instead of a singleTDI detector, if desired.

Non-Distorting Spectral Dispersion Systems

The present invention can be provided with a spectral dispersion filterassembly that does not convolve the image with the emission spectra ofthe light forming the image, thereby eliminating the need fordeconvolution of the emission spectra from the image. FIG. 27illustrates an embodiment of such a non-distorting spectral dispersionsystem 250 that employs a five color stacked wedge spectral dispersingfilter assembly 252. This embodiment is substantially similar to theembodiment shown in FIGS. 17, 18, and 19, except that spectraldispersing prism element 36 (of FIGS. 17, 18, and 19) is replaced byspectral dispersing filter assembly 252. The spectral dispersing filterassembly splits the light into a plurality of light beams havingdifferent bandwidths. Each light beam thus produced is directed at adifferent nominal angle so as to fall upon a different region of TDIdetector 44. The nominal angular separation between each bandwidthproduced by the spectral dispersing filter assembly 252 exceeds thefield angle of the imaging system in object space thereby preventingoverlap of the field images of various bandwidths on the detector.

Spectral dispersing filter assembly 252 comprises a plurality of stackeddichroic wedge filters, including a red dichroic filter R, an orangedichroic filter O, a yellow dichroic filter Y, a green dichroic filterCG, and a blue dichroic filter B. Red dichroic filter R is placed in thepath of collected light 34, oriented at an angle of approximately 44.0°relative to an optic axis 253 of collection lenses 32 a and 32 b. Lightof red wavelengths and above, i.e., >640 nm, is reflected from thesurface of red dichroic filter R at a nominal angle of 1°, measuredcounter-clockwise from a vertical optic axis 257. The light reflected byred dichroic filter R leaves spectral dispersing filter assembly 252 andpasses through imaging lenses 40 a and 40 b, which cause the light to beimaged onto a red light receiving region of TDI detector 44, which isdisposed toward the right end of the TDI detector, as shown in FIG. 27.

Orange dichroic filter O is disposed a short distance behind reddichroic filter R and is oriented at an angle of 44.5 degrees withrespect to optic axis 253. Light of orange wavelengths and greater,i.e., >610 nm, is reflected by orange dichroic filter O at a nominalangle of 0.5° with respect to vertical optic axis 257. Because theportion of collected light 34 comprising wavelengths longer than 640 nmwas already reflected by red dichroic filter R, the light reflected fromthe surface of orange dichroic filter O is effectively bandpassed in theorange colored region between 610 nm and 640 nm. This light travels at anominal angle of 0.5° from vertical optic axis 257, and is imaged byimaging lenses 40 a and 40 b so as to fall onto an orange lightreceiving region disposed toward the right hand side of TDI detector 44between a center region of the TDI detector and the red light receivingregion, again as shown in FIG. 27.

Yellow dichroic filter Y is disposed a short distance behind orangedichroic filter O and is oriented at an angle of 45° with respect tooptic axis 253. Light of yellow wavelengths, i.e., 560 nm and longer, isreflected from yellow dichroic filter Y at a nominal angle of 0.0° withrespect to vertical optic axis 257. Wavelengths of light reflected byyellow dichroic filter Y are effectively bandpassed in the yellow regionbetween 560 nm and 610 nm and are imaged by imaging lenses 40 a and 40 bnear vertical optic axis 257 so as to fall on a yellow light receivingregion toward the center of TDI detector 44.

In a manner similar to dichroic filters R, O, and Y, dichroic filters Gand B are configured and oriented so as to image green and blue lightwavebands onto respective green and blue light receiving regions of TDIdetector 44, which are disposed toward the left-hand side of the TDIdetector. By stacking the dichroic filters at different defined angles,spectral dispersing filter assembly 252 collectively works to focuslight within predefined wavebands of the light spectrum onto predefinedregions of TDI detector 44. Those of ordinary skill in the art willappreciate that the filters used in the spectral dispersing filterassembly 252 may have spectral characteristics that differ from thosedescribed above. Further, the spectral characteristics may be arbitraryand not limited to dichroic in order to achieve the desired dispersioncharacteristics.

The wedge shape of the dichroic filters in the preceding discussionallows the filters to be placed in near contact, in contact or possiblycemented together to form the spectral dispersing filter assembly 252.The angle of the wedge shape fabricated into the substrate for thedichroic filter allows easy assembly of the spectral dispersing filterassembly 252, forming a monolithic structure in which the wedge-shapedsubstrate is sandwiched between adjacent dichroic filters. If thefilters are in contact with each other or cemented together, thecomposition of the materials that determine the spectral performance ofthe filter may be different from those which are not in contact. Thoseof ordinary skill in the art will appreciate that flat, non wedge-shapedsubstrates could be used to fabricate the spectral dispersing filterassembly 252. In this case another means such as mechanically mountingthe filters could be used to maintain the angular relationships betweenthe filters.

In addition to the foregoing configuration, non-distorting spectraldispersion system 250 may optionally include a detector filter assembly254 to further attenuate undesired signals in each of the light beams,depending upon the amount of rejection required for out-of-band signals.FIG. 29 illustrates the construction of an exemplary detector filter 254corresponding to the five color bands discussed above and includes ablue spectral region 256, a green spectral region 258, a yellow spectralregion 260, an orange spectral region 262, and a red spectral region264, all of which are disposed side-by-side, as shown in the Figure. Thedetection filter assembly shown in FIG. 29 may be constructed bymounting separate filters in side-by-side arrangement on a commonsubstrate. Additionally, the ordinary practitioner in the art willunderstand that the filter may alternatively be placed at anintermediate image plane, instead of directly in front of TDI detector44.

In the embodiment shown in FIG. 27, light may pass through each dichroicfilter in the spectral dispersing filter assembly 252 twice beforeexiting the spectral dispersing filter assembly 252. This condition willfurther attenuate out-of-band signals, but will also attenuate in-bandsignals. FIG. 29 illustrates an eighth embodiment of the presentinvention in which the light does not pass through another dichroicfilter after reflection. In this embodiment, a plurality of cubedichroic filters, including a red cube filter 266, a yellow cube filter268, a green cube filter 270, and a blue cube filter 272 are spacedapart sufficiently to ensure that light does not pass through any of thecube filters more than once. As with the embodiment of FIG. 27, the cubedichroic filters are oriented at appropriate angles to image lightwithin a predefined bandwidth to distinct regions on a TDI detector 274.As the light is reflected from each of cube dichroic filters 266, 268,270 and 272, it is directed toward imaging lenses 40 a and 40 b, anddifferent bandpass portions of the light are focussed upon correspondingred, yellow, green, and blue light receiving segments or regions definedon a light receiving surface of TDI detector 274. If desired, anoptional detector filter assembly 276 of similar construction todetector filter assembly 254 (but without the orange spectral region)may be used to increase the rejection of out-of-band signals. It shouldbe apparent to those skilled in the art that separate spaced apartplate, or pellical beam splitters could also be used in this applicationinstead of the cube filters. In the eight embodiment illustrated in FIG.29, the image lenses 40 a and 40 b must be placed a sufficient distanceaway from the plurality of cube filters to minimize the clear aperturerequirement for lenses 40 a and 40 b. Those skilled in the art willappreciate the clear aperture in the plane orthogonal to the page canincrease as the distance between the lenses and plurality cube filtersincreases. Therefore, the placement of lenses 40 a and 40 b must bechosen to appropriately accommodate the clear aperture in both planes.

The foregoing descriptions of the last two additional preferredembodiments of imaging systems illustrate the use of four and five colorsystems. Those skilled in the art will appreciate that a spectraldispersing component with more or fewer filters may be used in theseconfigurations in order to constrict a system covering a wider or anarrower spectral region, or different passbands within a given spectralregion. Likewise, those skilled in the art will appreciate that thespectral resolution of the present invention may be increased ordecreased by appropriately choosing the number and spectralcharacteristics of the dichroic and or bandpass filters that are used.Furthermore, those skilled in the art will appreciate that the angles ororientation of the filters may be adjusted to direct light of a givenbandwidth onto any desired point on the TDI detector. In addition, thereis no need to focus the light in increasing or decreasing order bywavelength. For example, in fluorescence imaging applications, one maywish to create more spatial separation on the TDI detector between theexcitation and emission wavelengths by changing the angles at which thefilters corresponding to those wavelengths are oriented with respect tothe optic axes of the system. Finally, it will be clear to those skilledin the art that dispersion of the collected light may be performed onthe basis of non-spectral characteristics, including angle, position,polarization, phase, or other optical properties.

As with the earlier embodiments discussed above, these embodiments thatuse the filters will require that one or more light sources be used toprovide light that is incident on the object being imaged. Accordingly,various light sources disposed at different positions, Such as thoseshown in FIGS. 21–23 and discussed above, may be used to enhance theimage quality produced by each of these embodiments. For clarity and tosimplify the explanation of these embodiments, the light sources havebeen omitted in FIGS. 27 and 29; however, it will be recognized by thoseskilled in the art how such light sources may be employed in theseembodiments, based on the previous discussion of the use of the lightsources with respect to the earlier embodiments.

Although the present invention has been described in connection with thepreferred form of practicing it and modifications thereto, those ofordinary skill in the art will understand that many other modificationscan be made to the present invention within the scope of the claims thatfollow. Accordingly, it is not intended that the scope of the inventionin any way be limited by the above description, but instead bedetermined entirely by reference to the claims that follow.

1. A method for reducing crosstalk among a plurality of signals from aplurality of sources, each signal being assigned to a separate channeland primarily containing information corresponding to a different sourceamong the plurality of sources, comprising the steps of: (a) applyingspatial corrections to correct any misalignment of the signals betweenchannels, such that corresponding signals from different sources in theplurality of channels are aligned; and (b) for each channel,substantially reducing erroneous contributions to the signal assigned tothe channel from others of the plurality of signals.
 2. The method ofclaim 1, wherein the step of applying spatial corrections to correct anymisalignment of the signals between channels comprises the step ofapplying spatial corrections at a sub-pixel resolution.
 3. An article ofmanufacture adapted for use with a computer, comprising: (a) a memorymedium; and (b) a plurality of machine instructions, which are stored onthe memory medium, said plurality of machine instructions when executedby a computer, causing the computer to: (i) correct a signalmisalignment between a set of related signals to within sub-pixelresolution, wherein each one of the set of related signals primarilycontains information corresponding to a different specific source; and(ii) substantially reduce crosstalk contributions to each of the signalsfrom other of the signals in the set of related signals.
 4. An articleof manufacture adapted for use with a processor, comprising: (a) amemory medium; and (b) a plurality of machine instructions, which arestored on the memory medium, said plurality of machine instructions whenexecuted by a processor, causing the processor to: (i) correct a signalmisalignment between a set of related signals, wherein each one of theset of related signals primarily contains information corresponding to adifferent specific source; and (ii) substantially reduce crosstalkcontributions to each of the signals from other of the signals in theset of related signals.
 5. A system for reducing crosstalk among aplurality of related signals in a set, each one of the set of relatedsignals primarily containing information corresponding to a specificdifferent source from among a plurality of different sources,comprising: (a) a memory in which a plurality of machine instructionsdefining the parent application are stored; and (b) a processor that iscoupled to the memory to access the machine instructions, said processorexecuting said machine instructions and thereby implementing a pluralityof functions, including: (i) correcting a signal misalignment betweenthe plurality of related signals, each one of the plurality of relatedsignals in the set is substantially aligned with other of the pluralityof related signals in the set; and (ii) for each one of the plurality ofrelated signals in the set, reducing crosstalk contributions from otherof the plurality of related signals.